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(54) Remote control of a display system 

(57) A method and system for remote control of a 
target computer by a controller computer through a net- 
work. The graphic commands issued by the graphic 
engine of the target computer are "hooked" and trans- 
lated into a system independent format. The translated 
commands are then transmitted to the controller com- 
puter to be replayed on the display. The independency 
from the operating system is realized by defining a pro- 
tocol which is used to translate a subset of the graphic 
commands in the target computer before the transmis- 
sion to the controller computer where the commands 
are re-translated according to the same protocol. 



201, 



TARGET PC 



E03-/ 



GRAPHIC 




OISPUOT 




DISPLAY 


ENGINE 






DRIVER 





HOOK DISPLAY DRIVER — 209 



CONTROLLER PC 



DISPLAY 




DISPLAT 


DRIVER 





FIB. 2 

© 

00 

*r 

CO 
CO 

o 
Q_ 

UJ 

Printed by Xerox (UK) Business Services 
2.15.12/3.4 



EP 0 834 801 A2 



Description 

The present invention relates to remote control systems and particularly to a system and method for duplicating the 
screen image of a source data processing system on a screen of a destination system. 

5 Remote control programs are a special category of communication applications. They go beyond just connecting a 
computer with another computer to actually taking control of it. By connecting through phone lines and modems or 
through LAN connections, these programs can put all the resources of one data processing system (e.g. a personal 
computer or a workstation), usually called the target, under the complete control of another data processing system 
(e.g. a personal computer or a workstation, connected with a LAN), usually called the controller. The controller screen 

w duplicates the target one. The keyboard of the controller can enter characters directly on the target data processing sys- 
tem; the mouse of the controller can move the mouse pointer on the target screen. The controller screen constantly 
receives data from the target data processing system screen. Likewise, the keyboard of the controller sends data to the 
target data processing system every time a key is hit For all practical purposes, the controller user may as well be sit- 
ting in front of the target data processing system. 

75 This kind of remote control software can be useful in a number of circumstances. Just as an example an office per- 
sonal computer system can be controlled and operated from home personal computers. If the office PC is left con- 
nected to a modem, the control can be taken from the home PC and all the resources of the target PC can be controlled 
and used remotely. A more specific use is to dial in (via modem) to a network-connected PC at the office, log in, and 
run network applications or update shared files on the network. 

20 The bottleneck caused by modems can have a significant effect on the speed of remote control operations. To 
boost performances, remote control programs try to reduce and minimize the amount of data that needs to be transmit- 
ted over the communication lines (LAN or telephone cables). The refresh of the video image on the display screen of 
the target PC, which should be diplicated on the controller PC, constitutes the main point in the transmission speed, 
due to the big quantity of information which has to be transferred. In the past, remote control programs worked by send- 
as ing and refreshing the entire image of the target PC screen to the remote PC every fraction of a second. The constant 
rate at which the screen images were sent to the controller PC was called the screen refresh rate. The controller PC 
screen accurately reflected the target PC screen as changes at the target PC would be included in the next screen 
refresh. This approach was, of course, not very satisfactory in term of speed, because of the enormous amount of data 
to be transmitted at each refresh. 

30 Often, a change of the image is limited to a small section of the screen (e.g. moving the mouse pointer, pulling down 
a menu, highlighting an object) and only the changed data need to be refreshed. The controller system can use the 
screen data previously received and just refresh the modified part of the screen. By cutting out redundant screen 
refresh data, a remote control program can improve the response time. Cache memories speed the performance of 
hard drives and memory chips, and can also be useful for remote control programs. The idea behind a cache is to keep 

35 recently used data in a temporary storage area that is quickly accessible. If the data are soon needed again r rt is much 
quicker to access them from a fast-access temporary storage area than to retrieve from the original source. According 
to this technique, elements of a recently displayed screen are saved as a cache in extended memory. The entire page 
redraws from the controller PC screen cache, eliminating the need to transfer large amounts of information over the 
modem from the target to the controller PC. Screen elements that are often redrawn from a screen cache include bit 

40 maps, text and fonts. 

US patent 5,491 ,780 describes a technique for reducing the transmission of bit-map images, by limiting the trans- 
mission and the refreshing only to the portion of screen really affected by the graphic commands of the target computer. 

Anyway the above techniques, while reducing the amount of image data to be sent over the communication lines, 
cannot avoid the transmission of image portion usually in the form of bit maps each time the image is modified. This 
45 transmission still represents the bottleneck of the remote control programs, because of the low speed at which the bit 
map with the modified area can be sent through a modem or a LAN; this burden of course can be very heavy in case 
of a large modification which requires a large bit map. 

It is an object of the present invention to provide a technique which overcomes the above drawbacks. 

According to the invention we provide a method for duplicating and refreshing, on at least one destination data 
so processing system a graphic image displayed on a source data processing system, the at least one destination data 
processing system comprising: 

a first display unit for displaying graphic images and 

a first display driver for modifying the image on the first display unit; 

55 

the source data processing system comprising: 
a second display unit for displaying images, 
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- a graphic engine far generating a plurality of graphic commands and 

- a second display driver for modifying the image on the second display unit according to the graphic commands gen- 
erated by the graphic engine; 

5 the at least one destination data processing system and the source data processing system being connected through 
a network, the method comprising the steps of: 

- intercepting the graphic commands generated by the graphic engine; 

- passing the graphic command to the second display driver; 

w - translating a subset of the graphic commands into an operating system independent format, according to a prede- 
fined protocol; 

- sencfing the system independent graphic commands, through the network, to the at least one destination data 
processing system; 

- transforming the system independent graphic commands into the at least one destination data processing system 
15 format, according to said protocol; 

- passing the transformed graphic commands to the first display driver. 

Furthermore, according to the present invention we provide a system for duplicating and refreshing, on at least one 
destination data processing system a graphic image displayed on a source data processing system, the at least one 
zo destination data processing system comprising : 

- a first display unit for displaying graphic images and 

- a first display driver for modifying the image on the first cfisptay unit; 

25 the source data processing system comprising: 

- a second display unit for displaying images, 

a graphic engine for generating a plurality of graphic commands and 

- a second display driver for modifying the image on the second display unit according to the graphic commands gen- 
30 erated by the graphic engine; 

the at least one destination data processing system and the source data processing system being connected through 
a network, the system comprising: 

35 - means for intercepting the graphic commands generated by the graphic engine; 

- communication means for passing the graphic command to the second display driver; 

- means for translating a subset of the graphic commands into an operating system independent format according 
to a predefined protocol; 

- network means for sencfing the system independent graphic commands to the at least one destination data 
40 processing system; 

- means for transforming the system independent graphic commands into the at least one destination data process- 
ing system format, according to said protocol; 

- communication means for passing the transformed graphic commands to the first display driver. 

45 Various embodiments of the invention will now be described in detail by way of examples, with reference to accom- 
panying figures, where: 

Fig. 1 shows schematically the network and the personal computers which may be utilized to implement the method 
and system of the present invention; 
so Fig. 2 shows schematically a preferred embodiment of the present invention; 

Fig. 3 shows schematically the method steps according to a preferred embodiment of the present invention; 
Fig. 4 shows schematically an embodiment of the present invention. 

Fig. 1 illustrates an example of a typical Local Area Network (LAN) where a remote control system can be imple- 
55 merited. It comprises a LAN server 101 which acts as a communication bridge between two or more data processing 
systems. In other connection systems the function of the server would be performed, for example, by a bridge, a 
modem, a cable or, in general, any device that interposed between two or more data processing system, makes them 
connected. 
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The data processing system 105 is the controller and can alternatively take the control of one of the other data 
processing system 103, 107 and 109. 

Data processing systems 103, 105, 107 and 109 can be personal computers or workstations available on the mar- 
ket They preferably include a display unit and a keyboard, coupled in a manner well known in the art. Additionally, the 
5 data processing systems include a processor system unit which may serve to mount a fixed disk drive and a dskette 
drive in addition to the main processor and memory. Further, in order to facilitate the utilization of a graphical user inter- 
face, the data processing systems preferably include a graphic pointing device, such as a mouse, which may be utilized 
to manipulate the position of a pointer within a visual display on the screen. 

Those skilled in the art will appreciate that data processing systems described above may be implemented utilizing 
10 any state-of-the-art personal computer, such as the PS/2 Personal Computer manufactured by International Business 
Machines Corporation. (PS/2 is a trademark of International Business Machines Corporation) 

Data processing systems are usually provided with a graphical user interlace (GUI) to allow a user to control the 
data processing system and to present the results of user actions on the system display. In a graphical user interface, 
applications and data are generally presented as objects depicted on a user interface. A user is then provided with a 
is graphical, intuitive interface to a complex data processing system which permits graphic selection of depicted objects 
and manipulation of applications corresponding to those objects. 

There are a number of different graphical user interface environments commercially available which utilize the 
arrangement described above. These environments include the System 7 operating system developed by Apple Com- 
puter, Inc. (System 7 is a trademark of Apple Computer, Inc.), the Windows graphical user interlace developed by the 
20 Microsoft Corporation (Windows is a trademark of Microsoft Corporation) and the OS/2 Presentation Manager devel- 
oped by International Business Machines Corporation (OS/2 and Presentation Manager are trademarks of International 
Business Machines Corporation). 

The window environment is generally part of the operating system software that includes a collection of utility pro- 
grams for controlling the operation of the computer system 100. 
25 In the example shown in Fig. 1 the control is over the system 103, which is defined the target system and all the 
resources of system 103 can be used and activated by the controller system. For performing such control, system 105 
must be able to represent on its display an exact copy of the target display 103. 

With reference to Fig. 2 a preferred embodment of the present invention is shown. Under state of the art graphical 
operating system, an application can display or modify images on the screen through a set of device independent APIs 
30 provided by the operating system graphic engine (203). The graphic engine (203) performs graphic operations by call- 
ing entry points in the display driver (205) that performs the appropriate operations on the driver for which the driver is 
written. The driver is the end point of the operation since it modifies the graphic frame buffer or the display (207) that is 
managed directly by the hardware. 

According to the present invention another module is inserted in the system. The hook display driver (209) inter- 
35 cepts the calls issued by the graphic engine (203) to the display driver (205). Then the hook display driver (209) dupli- 
cates the graphic commands and send them both to the target display driver (205) and to the controller cfisplay driver 
(213) in order to realize a correspondence between the image displayed on the target system and on the controller sys- 
tem. 

Obviously, this solution is much more efficient in terms of speed of transmission than the method of sending a bit 
40 map of the target image. 

However, while the bit-map is a sort of "universal format" which can be understood and processed by any operating 
system, the transmission of a set of commands can create some problems in terms of compatibility if the operating sys- 
tem of the target and the controller are not identical. 

To realize this compatibility, the hook display driver (209) according to the present invention, translates the graphic 
45 commands issued by the graphic engine (203) of the target computer into a system independent format using a prede- 
fined protocol. The system independent commands are then transmitted through the network to the controller compu- 
ter, where they are re-translated in graphic commands for the display driver (213) using the same protocol. 

According to a preferred embodiment of the present invention, only a subset of the graphic commands issued by 
the graphic engine are translated and transmitted to the controller. The reason is that a limited set of commands covers 
so the most of the operations performed by a display driver. 

The following is a definition of the constants and data structures expressed in the 'C* language as used in a pre- 
ferred embodiment of the present invention: 
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List of constants and data structure expressed in 'C language used in the protocol: 
Command identifier values: 

#define ACCHjCMD.BITBLT 1 

#define ACCM_CMD JB ITBLT_S AVE 2 

#define ACO!_CMD_BITBLT_HIT 3 

#define ACCM_CMD_BITBLT__SRCCOPY 4 

# define ACCM_CMD_S AVBSCREENB I1MAP__SAVE 5 
# define ACCK_CMD_SAVESCREENBITMAP_RESTORE 6 

#def ine ACCM_CMD_B ITBLT_REPEAT 7 

#define ACCM_CMD_TEXTOUT 8 

Brush field values: 

#define DIC_BRUSH_SOLID 0 

#define DIC_£RUSH_HATCHED 2 

#define DIC_BRUSH_PATTERN 3 

#define DICJBRUSH_SIMULATED.PATTERN 4 

Extended Text Out field values: 



25 
30 
35 
40 
45 
50 
55 



5 



EP 0 834 801 A2 



Background field values: 



#define BK^TRANS PARENT 1 

#def ine BK^_OPAQUE 2 

Font identifier field values: 

#define F0NT_C0URIER 1 

#define FONT_HELV 2 

fdefine FONT_SYSTEM 3 

#define FONT_TMSROMAN 4 



typedef struct tagDCAF _B ITMAP INFOHEADER 
{ 

unsigned short usSize; 
unsigned short usBmpWidth; 
unsigned short usBnpHeight ; 

unsigned char bPlanes; 
unsigned char bBit count; 

} DC AF__B ITMAP INFOHEADER ; 
typedef DCAF_JB ITMAP INFOHEADER * PDCAFJ3 ITMAP INFOHEADER ; 



typedef struct tagDCAF_BITMAPINFO 
{ 

DCAF_B ITMAP INFOHEADER dc fbnpin f oheader ; 
unsigned char bBmpBytes[MAX_BRUSK_BITMAP_SIZE] ; 

} DC AF__B ITMAP INFO ; 
typedef DC AF_B ITMAP INFO *PDCAF_B ITMAP INFO; 

typedef struct tagDCAF_BRUSHINFOHEADER 
{ 

short sSize; 
short sType; 
long lColor; 
short s Hatch; 

long iBkColor ; 

short usXalign; 
short usYalign; 
) DCAF_BRUSH INFOHEADER ; 
typedef DCAF — BRUSH INFOHEADER * PDCAF__BRUSH INFOHEADER ; 



typedef struct tagDCAF_JJRUSHINFO 
{ 

DCAF_BRUSH INFOHEADER dc f brushinf oheader ; 
DCAF_B ITMAP I NFO dcf tap info; 
) DCAF_ BRUSHINFO ; 
typedef DCAF__BRUSH INFO *PDCAF_BRUSHINFO; 

typedef struct tagDCAF_BITMAPSAVER 
{ 

unsigned short usBmpSize; 
unsigned short us Widths- 
unsigned short usHeight; 
unsigned short usWidthBytes ; 
unsigned short us Plane s; 
char tap; 
) DCAF^B I TMAPS AVER ; 
typedef DCAFJITMAPSAVER * PDCAF_BITMAPS AVER ; 

typedef struct .DCAFRECT { 

unsigned short usRectXLeft; 

unsigned short usRectYBottom; 

unsigned short usRectXRight ; 

unsigned short usRectYTop; 

) DCAFRECT; 
typedef DCAFRECT *PDCAFRECT; 
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typedef struct _ACCH_TEXTOUT { 
unsigned char bCommandType ; 
unsigned short usCmdLength; 
unsigned short ueEtoOptions; 
char bClipPresent; 
DCAFRECT rectClipArea; 
char bOpaquePresent ; 
DCAFRECT rectOpaqueArea; 
unsigned short usDestX; 
unsigned short usDestY; 
unsigned short usCount; 
unsigned short usSel Height; 
unsigned short usSelWidth; 
unsigned short us Points ize; 
unsigned short usFaceName; 
unsigned long ulBkColor; 
unsigned long ul Text Co lor; 
unsigned short usBkMode; 
unsigned short usWeight; 
unsigned short usWidth; 
unsigned short usHeight; 
char cltalic; 
char cUnderline; 
char cStrikeOut ; 
char cText ; 

) ACCHJTEXTOOT; 
typedef ACCH_TEXTOUT *PACCH_TE3CT0UT; 



25 typedef struct _ J ACCM_B ITBLT_CORE { 

unsigned char bCommandType ; 

unsigned short usCmdXength; 

unsigned short usDestXOrigin; 

unsigned short usDestYOrigin; 

unsigned short usSrcXOrigin; 

unsigned short usSrcYOrigin ; 
30 unsigned short usXExtension; 

unsigned short usYExtension; 

long lRop3 ; 

) ACCWLBITBLT_CORE; 
typedef ACCH_BITBLT — CORE *PACCH_BITBLT_CORE; 

35 typedef struct _ACCHJB ITBLT { 

ACCHJB ITBLT__CORE bop_core; 
DCAF_3RUSHINF0 Brush; 

> ACCWL3ITBLT; 

40 typedef ACCM_BITBLT * PACCMLBITBLT ; 

typedef struct _ACCH_B ITBLT_S AVE { 
ACCM_BITBLT_CORE bmp_core; 
unsigned char bBmpID; 
DCAF_B ITOAPS AVER bmpSaver; 
45 ) ACCH_B ITBLT_S AVE ; 

typedef ACCMJBITBLTjSAVE *PACOt_3ITBLT_SAVE; 

typedef struct _ACCH_BITBLT_HIT { 
ACCH_BITBLT_CORE bnp.core; 
unsigned char bBmpID; 
) ACCH_BITBLT_HIT; 
50 typedef ACCH_BITBLT_HIT *PAOCH^BITBLT^IT; 

typedef struct _^ACCM__SAVESCREENBITMAP { 
unsigned char bCommandType ; 
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usCmdLength; 
usRectXLeft; 
usRectYBottom; 
usRectXRight; 
usRectYTop; 
) ACCMLS AVESCREENBITMAP ; 
typedef ACCM__SAVESCREENB ITMAP * PACO^S AVESCREENB ITMAP ; 



unsigned short 
unsigned short 
unsigned short 
unsigned short 
unsigned short 
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The information necessary to replay a graphical command are stored in a structure which contains the command 
identifier and the parameters used to replay the command. For example when a text is drawn the information regarding 
this event is stored in the structure ACCM_TEXTOUT whose first field contains the value ACCM_CMD_TEXTOUT and 
the other fields are used to replay the text 

75 The protocol described above is believed to handle most of the events that modify the display on any known oper- 
ating system. For example, using the Microsoft Windows operating system the above described protocol is able to hook 
and replay an average of 95% of the screen changes. 

Fig. 3 shows the steps of the method according to a preferred embodiment of the present invention. At step 301 the 
server 101 connects the target computer 103 with the controller computer 105. At step 303 the hook display driver 209 

20 intercepts the graphic commands issued by the graphic engine 203 to the display driver 205. Step 305 identifies those 
graphic commands belonging to a predefined subset of commands and extracts the related parameters. Step 307 
translates the graphic commands identified by step 305 and the related parameters into a system independent format 
according to the predefined protocol described above. Step 309 sends the translated commands and parameters to the 
display driver 213 of the controller computer, where the commands will be retranslated from the system independent 

25 format into the controller system format to be reproduced on the controller display 215. 

Those skilled in the art will appreciate that a proper translation tailored on the format of each system (source or 
controller) must be implemented in order to realize the correspondence between two different systems and the common 
protocol. However, due to the limited number of commands belonging to the subset, this additional operations do not 
constitute a substantial overhead neither in the functioning of the system nor from the programming point of view. 

so According to a preferred embodiment of the present invention the complete correspondence of the controller dis- 
play with the target display can be granted by sending every predetermined period of time (eg. 5 sec.) a bit-map of the 
image actually displayed on the target display to "synchronize" the controller display in case of events not covered by 
the protocol. 

To check whether the image replayed on the controller display is coincident with the target display, a "shadow" bft- 

35 map copy of the target display is maintained in the storage of the target computer. Each time a command issued by the 
graphic engine is translated using the protocol and transmitted to the controller computer to be re-translated and 
replayed, the same operation is done on the "shadow" bit-map image: the system independent commands are re-trans- 
lated and operated on the "shadow" bit-map. Every predetermined period of time (5 sec. in the preferred embodiment) 
the "shadow" image is compared with the actual target display and if any difference is detected, the portion of the 

40 screen to be modified is transmitted in the bit-map format to the controller computer. 

Fig. 4 shows schematically the "synchronization" mechanism: a subset of the graphic commands, issued by the 
graphic engine 203 to the display 207 in the target PC, is intercepted and translated by translator 403 and sent to the 
controller PC where they will be re-translated and reproduced on display 215. This last operation is simulated also on 
the "shadow bit-map" 401 : in this way the target PC can always check what is actually displayed on controller display 

45 21 5. Every predetermined period of time (eg. 5 seconds) the shadow bit-map is compared with the bit-map of the dis- 
play 207 and where a difference is detected a "synchronization" is performed: the bit-map is transmitted to the controller 
display 21 5 to update the cfisplayed image and re-establish the correspondence between the two images. 

The bit-map transmission, as mentioned above, is a heavy and time consuming activity; but according to the syn- 
chronization mechanism described above it is performed only once every period of time and only when actually needed. 

so In this way the high reliability of this kind of transmission is obtained while not affecting the speed of the communications 
between the systems. 



Claims 



55 1 . A method for duplicating and refreshing, on at least one destination data processing system a graphic image dis- 
played on a source data processing system, the at least one destination data processing system comprising: 



- a first display unit for displaying graphic images and 
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- a first display driver for modifying the image on the first display unit; 
the source data processing system comprising: 

- a second display unit for cfisplaying images, 

- a graphic engine for generating a plurality of graphic commands and 

- a second display driver for modifying the image on the second display unit according to the graphic commands 
generated by the graphic engine; 

the at least one destination data processing system and the source data processing system being connected 
through a network, the method comprising the steps of: 

- intercepting the graphic commands generated by the graphic engine; 

- passing the graphic command to the second display driver; 

- translating a subset of the graphic commands into an operating system independent format, according to a 
predefined protocol; 

sending the system independent graphic commands, through the network, to the at least one destination data 
processing system; 

- transforming the system independent graphic commands into the at least one destination data processing sys- 
tem format according to said protocol; 

passing the transformed graphic commands to the first display driver. 

The method of claim 1 further comprising the steps of: 

copying in the storage of the source data processing system a bit-map of the image displayed on the second 
display unit; 

modifying the bit-map image according to said subset of graphic commands; 

- comparing, every predetermined period of time, the bit-map image with the image displayed on the second dis- 
play unit; 

- if the difference between the bit-map image and the image displayed on the second display unit exceeds a pre- 
determined threshold, updating the bit-map image and sending, through the network, the bit-map image to the 
at least one destination data processing system. 

The method of any preceding claim wherein the network is a Local Area Network. 

A remote control system using the method of any preceding claim. 

A system for duplicating and refreshing, on at least one destination data processing system a graphic image dis- 
played on a source data processing system, the at least one destination data processing system comprising : 

a first display unit for displaying graphic images and 

a first display driver for modifying the image on the first display unit; 

the source data processing system comprising: 

a second display unit for displaying images, 

- a graphic engine for generating a plurality of graphic commands and 

a second display driver for modifying the image on the second display unit according to the graphic commands 
generated by the graphic engine; 

the at least one destination data processing system and the source data processing system being connected 
through a network, the system comprising: 

- means for intercepting the graphic commands generated by the graphic engine; 

- communication means for passing the graphic command to the second display driver; 

- means for translating a subset of the graphic commands into an operating system independent format, accord- 
ing to a predefined protocol; 

network means for sending the system independent graphic commands to the at least one destination data 
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processing system; 

- means for transforming the system independent graphic commands into the at least one destination data 
processing system format according to said protocol; 

- communication means for passing the transformed graphic commands to the first display driver. 

5 

6. The system of claim 5 further comprising: 

- timer means tor counting a predetermined period of time; 

- means for copying in the storage of the source data processing system a bit-map of the image displayed on the 
10 second display unit; 

means for modifying the bit-map image according to said subset of graphic commands; 

means for comparing, every predetermined period of time, the bit-map image with the image displayed on the 

second display unit and for updating the bit-map image if the difference exceeds a predetermined threshold; 

- network means for sending the bit-map image to the at least one destination data processing system, if the drf- 
15 ference between the bit-map image and the image displayed on the second display unit exceeds a predeter- 
mined threshold. 

7. The system of claim 5 or 6 wherein the network is a Local Area Network. 
20 8. A remote control system implementing the system of claim 5, 6 or 7. 
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